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ABSTRACT 

The usefulness of response latency data for 
biographical inventory items was assessed for improving the 
inventory's validity. Focus was on assessing whether weighting item 
scores on the basis of their latencies improves the predictive 
validity of the inventory's total score. A total of 120 items from 
the Armed Services Appliceuit Profile (ASAP) were administered via 
computers to 1,090 male Navy recruits at the Recruit Training Center 
in San Diego (California); the regular score, latency -weighted 
scores, and measures of deviant latencies were obtained. The 
latency-weighted scores did not improve the ASAP's validity in 
predicting 6-month retention beyond that obtainevl with the regular 
score; the deviant latency measures did not function as suppressor or 
moderator variables to increase the ASAP's validity. Subgroups ox 
items with differing latencies varied systematically in their 
internal-consistency reliability (with increased reliability for 
subgroups with shorter latencies), and a snail subgroup of items with 
moderate latencies was almost as valid as the regular score, 
suggesting that latency data may be useful in writing and selecting 
inventory items. Five tables and four cxraphs complement the study. A 
34-item list of references is included. (Author/SLD) 
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Abstract 

This study assessed the usefulness of response latency data for 
biographical inventory items in enhancing the inventory's validity. The Armed 
Services Applicant Profile (ASAP) was computer administered to Navy recruits, 
and the regular score, latency-weighted scores, and measures of deviant 
latencies were obtained. The latency-weighted scores did not improve the 
ASAP's validity in predicting six-month retention, when used instead of or in 
addition to the regular score, and the deviant latency measures did not 
function as suppressor or moderator variables to increase the ASAP's validity. 
But subgroups of items with differing latencies varied systematically in their 
internal-consistency reliability (with increased reliability for subgroups 
with shorter latencies) ^ and a small subgroup of items with moderate latencies 
was almost as valid as the regular score ^ suggesting that latency data may be 
useful in writing and selecting inventory items. 
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Response Latency Measures for Biographical Inventories 
Recent theoretical and empirical work in personality and social 
psychology, coupled with the advent of computerized testing , raises the real 
possibility of improving the validity of personality, interest, and 
biographical inventories by administering theni via computer and using 
information about latency of responding to the items to modify conventional 
scoring techniques. 

Response latencies on personality inventory items and personality-trait 
adjectives have been extensively studied since the 1970s, A key finding is 
that items with long latencies are unstable: the responses to these items 
tend to change on retest. In itemraetric studies, latencies and the proportirn 
of changed responses (over a four-week interval) correlated .21 to .41 for 
MMPI (Hathaway & McKinley, 1951) items (Dunn, Lushene, & O'Neil, 1972), 
latencies and changed responses (over a one*week period) correlt^ted .36 for 
Personality Research Form (PRF; Jackson, 1984) items (Holden, Tekken, & 
Jacksoii, 1985), and latencies and changed responses (over p one-month period) 
correlated .49 for Basic Personality Inventory (BH; Jackson, 1989) items 
(Holden & Fekken, 1990).^ (But in an itemmetric study that used changed 
responses on immediate retest ing, latencies and the Ambdex index [Goldberg, 
1963), a measure of instability, correlated -.05 [not significant] for PRF 
items [Rogers, 1973].) In experiments on individual differences, the PRF 
items that each subject changed on retest (immediately in one experiment; 
after a one-week period in the other) were predicted significantly better than 
chance on the basis of which items had the longest latencies for him or her 
during the initial administration (Fekken & Jackson, 1988). 

Several otherwise divergent conceptualizations are alike in suggesting 
that long latencies for inventory items reflect difficulty in responding. 
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Some of the conceptualization are based on item characteristics, and others on 
the interaction between individual differences and itam characteristics 
(Fekken & Jackson, 1988). The item characteristic conceptualizations argue 
that the difficulty comes about because the item is hard to understand — 
unreadable, ambiguous, etc. (e.g., Dunn et al . , 1972; Hanley, 1962), The 
conceptualizations concerned with the interaction between individual 
differences and item characteristics contend that the difficulty arises 
because (a) the person has trouble in applying the item to himself or herself- 
-the item may deal with matters that are unfamiliar or unknown to the person, 
or the different response alternatives to the item may appear equally 
descriptive of him or her (e.g., Kuncel, 1973; Markus, 1977; Rogers, 1974a, 
1974b) or (b) the items may arouse emotions (e.g., Gilbert, 1967; Temple & 
Geisinger, 1990). 

The observed link between the response latency of personality items and 
the items* instability implies that the items' latencies are also associated 
with the items* validity, especially their predictive validity — the items with 
the longest latencies not only being the least stable but also the least valid 
over time. The findings ii the two investigations that bear on this issue are 
inconsistent. In one study (Holden et al., 1985), latencies for PRF items 
correlated -.22 with a concurrent validity criterion (a composite of self- 
ratings, self-reports on an adjective checklist, and preference ratings), but 
in a second investigation (Holden & Fekken, li90), latencies for BPI items 
correlated -,11 (not significant) with another concurrent validity criterion 
(ratings by clinicians). However, the findings of these studies may be 
affected by the dlchotomous format of the items: items with extreme 
endorsement proportions tend to be more stable and less valid (Goldberg, 
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1963), Furthermore, these results were based on concurrent criteria, and the 
consequences of the items' instability would be more pronounced If predictive 
criteria were used* 

The purpose of the present study was to determine whether the findings 
about the connection between Item latencies and Item instability can be used 
to improve an inventory's validity. More specif Ic/'lly, the main goal was to 
assess whether weighting item scores on the basis of their latencies improves 
the prcUit:tlve validity of the Inventory's total score. A secondary aim was 
to assess whether measures that reflect the extent to which subjects' 
latencies are deviant function as suppressor or moderator variables to 
increase the validity of the inventory's total score. The notion is that 
de^'lant latencies reflect an unusual pattern of responding to the inventory, 
stemming from idiosyncratic difficulties with certain Items, poor test-taking 
attitudes, and other variables that attenuate validity. Hence using measures 
of deviant latencies to suppress this invalid variance or to exploit their 
interaction with the inventory's score should increase validity. 

Method 

Overview 

One hundred and twenty items from the Armed Services Applicant Profile 
(ASAP), a biographical inventory, were computer administered to Navy recruits 
(all men), and the subjects' response choices and respcnse latencies were 
recorded. The regular score for the ASAP and three kinds of latency-weighted 
scores were obtained: (a) regular scores for .ubgroups of items with 
different latencies, for optimal weighting by standard multiple-regression 
methods; (b) item scores directionally welghtea by their latencies (i,e., less 
weight for items with relatively long latencies); and (c) item scores 
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nondirectionally weighted by their latencies (i.e., less weight for items with 
either relatively long or relatively short latencies). Measures of deviant 
latencies and, for exploratory purposes other suppressor/moderator variables 
were also obtained. Data for the criterion, retention in the Navy for six 
months, was subsequently secured. 

Background . The ASAP (Trent, Quennette, & Pass, 1989) is designed to 
predict the adjustment of enlisted personnel to military service. The final 
version of the ASAP consists of two 50-item alternate forms drawn from an 
initial pool of 170 heterogeneous items chosen for their potential relevance 
to adjustment. The items have three to five alternatives, and the 
alternatives are separately scored with weights of 1 to 3 that have been 
empirically derived to predict retention at 21 months of service. The items 
encompass six factors (nondelinquency , work orientation, work ethic, academic 
achievement, social adaptation, and athletic involvement). 

The inventory's predictive validity against retention criteria has been 
extensively studied, using a cohort of applicants for active duty in all the 
armed services. The present form of the ASAP or earlier forms (with 50 to 130 
Items) correlated •IS to .20 with retention at 6 months (T, Trent, personal 
communication, August 1986), .21 with retention at 21 months (Trent et al., 
198^^, and .27 with retention at 36 months (Trent, 1989), The internal- 
consistency reliability was .71 to .76 for 50-ltem forms (Trent et al., 1989, 
1990) and ,77 for 125 Item forms (T. Trent, personal communication, December 
1987) . 

It ems > A set of 120 Items was available for this study. Twenty-six of 
the other 50 Items In the Initial pool had been dropped previously because 



they concerned circumstances beyond the respondent's control, they might 
involve ethnic or social class bias, they were intrusive, or they asked about 
the type of high school credential (T. Trent, personal communication, October 
27, 1988). The additional 24 items were eliminated for this study because 
they duplicated remaining items. 

Minor editorial changes were made in the 120 retained items to achieve a 
consistent format and to eliminate unnecessary instructions (e.g., "Pick the 
main one"), and the items were arranged in random order. 

Because the current item weights for the ASAP are unavailable for some 
of the 120 items and are based on retention for 21 months rather than the six- 
monLh period used in this study, new item weights were obtained, using the 
same procedures and the same cohort data (JI-13,172 to 26,857) employed in 
deriving the current weights (Trent et al., 1989; M. A. Quennette, personal 
communication, January 1989), but for a six-month period. 

In brief, a modification of the "horizontal percent" method (Stead & 
Shartle, 1940) for deriving empirical weights for biographical items was 
employed. The percentage of applicants retained was computed for each 
alternative for the 90 items common to the two original 130 item forms of the 
ASAP. The distribution of these percentages was trichotomized, and 
alternatives with percentages in the top third were given a weight of 3, 
alternatives in the middle third a weight of 2, and those in the bottom tliird 
a weight of 1. These weights were also given to the alternatives for the 
items unique to each form. An exception was made for alternatives indicating 
th^ the respondent did not graduate from high school: these alternatives 
were assigned a weight of 1, regardless of their actual weight, to make the 
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ASAP independent of high school diploma status for policy reasons. (Only one 
of the 120 items had an alternative that was altered for this reason.)^ 

Computer administration . The paper-and-pencil version of the ASAP 
was adapted for computer administration via the same Hewlett Packard Integral 
Personal Computer used in the Accelerated Computerized Adaptive Testing — Armed 
Services Vocational Aptitude Battery system (Tiggle & Rafacz, 1985). The 
computer-adapted version of the ASAP was designed to be as close as possible 
to the original one in all important respects. 

The computer keyboard was simplified, consisting of numeiJical keys for 
entering the subjecc's identification number; keys labeled A, C. D, and E 
for response choices; an Enter key; and a Help key. The subject chose a 
response and recorded it by pressing the Enter key. The response could be 
changed at will before the Enter key was pressed. After the Enter key was 
pressed, the next item was presented, and the subject could not return to the 
previous item or earlier ones. The subject could seek assistance from the 
proctor by pressing the Help key and raising his hand. 

The pertinent instructions folic;: 

Read each question and all of its possible answers carefully, then 

select the pT>^ answer that is best or most appropriate for you.... You 

should work quickly but be as accurate as you can* Your answers to som e 

of these questions mav be verified for accuracy and honesty . 

Subjects were not informed that their latencies were being recorded. 

The following information was recorded for each item; 

1. The response choice. 

2. The number times that the response was changed. 
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3. The latency (in hundredths of a second) between the time that the 
item was presented and the Enter key was pressed. 

4, The number of tiroes that the Help key was used. 

ASAP measures. The regular ASAP score (the sum of the regular item 
scores) and three kinds of latency-based ASAP scores were secured. The 
latency-based scores employed standardized latencies » and two versions of each 
score were obtained. (Items for which the subject used the Help key were 
excluded in standardizing the latencies and in the latency-based scores; items 
for which the subject changed his responses were included in the 
standardization and in the scores because of the prevalence and relevance of 
changed items. ) 

One version of the scores, using double-standardized latencies to 
eliminate the main effects of individuals and items (e.g., Popham & Holden, 
1990) J reflected conceptualizations concerned with the interaction between 
individual differences and item characteristics. 

First, items were standardized to eliminate item differences associated 
with readability, ambiguity, and other characteristics. For this purpose, 
"interquartile deviations" were computed: (Actual Latency-Sample Median)/ 
Sample Interquartile Range. This nonparametric procedure (Tukey, 1977) was 
employed, instead of the conventional procedure, to reduce the effects on the 
standardization of the extrerre skewness in the latency data. Note that this 
linear transformation does not distort the real-time character of the latency 
data (Pachella, 1974). 

Then, using these interquartile deviations, each subject's latencies 
were standardized to eliminate individual differences associated with reading 
speedy reaction time, and similar characteristics. For this purpose, "double- 
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standardized Interquartile deviations" were computed: (Interquartile 
Deviation-Subject *s Median)/Subject's Interquartile Range. 

T>^e other version of the scores, employing single-standardized latencies 
to eliminate the main effects for individuals, reflected conceptualizations 
linked with item characteristics. Us,' ig the actual latencies, each subject's 
latencies were standardized to eliminate individual dif ferencf^s . 
"Interquartile deviations" were computed: (Actual Latency-Subject's 
Median)/Subject 's Interquartile Range. 

The three latency-based ASAP measures follow: 

1. Item subgroup scores: mean regular item score for each of 10 
subgroups of 12 ^ terns, the subgroups varying in their latencies, and the items 
in the subgroups differing from subject to subject. For example, Subgroup 1 
had the items with the largest interquartile deviations (the longest 
latencies) for each subject, and Subgroup 10 the items with the smallest 
interq\«artile deviations (the shortest latencies). When an item was excluded 
for a subject because the Help key was used, his Subgroup 10 had the 11 Items 
with the smallest interquartile deviations.) 

Ten groups of items were "9ed to achieve adequate reliability while 
permitting an examination^ of subsets of items with extreme latencies. The 
scores were intended to be combined by multiple regression methods that weight 
the scores for maximum validity in ^.redicting the retention criterion. 

2. Direct ionally weighted item score: the mean of the item scores that 
are directionally weighted by their items' corresponding latencies- Each 
subject's items were classified into nine categories on the basis of the 
interquartile deviations, ranging from -.81 or more (the shortest latencies) 
to 2.00 or more (the longest latencies). His regular score for each item was 
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shrunk towards the mean for the sample, depending on the extremeness of the 
item's latency (items with very large interquartile deviations, indicative of 
long latencies, were shrunk the most) and the distance between his score and 
the sample mean. (See Table 1.) The subject* s shrunken item score was 
calculated as follows: Subject *s Item Score ^ Shrinkage Rate (Sample Mean - 
Subject's Item Score). For example, suppose a subject had an interquartile 
deviation of -.2 and a score of 1 for an item, and the mean item score was 3. 
His directionally shrunken score would be 1 + 25% (3 1) - 1.5. 

Nine categories were chosen to provide a sufficient range of adjustments 
in the item scores. The nine represent equal intervals for the interquartile 
deviations (except for the intervals at each end — the interval of 2.00 or more 
at the hij,h end corresponds to an "outside" outlier; Tukey, 1977),^ The rates 
of shrinkage for the intervals were in equal steps, going from 100% for the 
lowest interval to 0% for the highest. The basic rationale for this weighting 
procedure is that the score is invalid for an item with a very long latency, 
and hence the best estimate of this score is the sample mean. Thus the longer 
the latency, the more the item's score is shrunk to the mean. 

3 Nondirectionilly weighted item score: the mean of the item scores 
that are nondirectionally weighted by their lattacies* This score was 
computed in the same way as the directional score, except that items with 
latencies at either extreme (very large interquartile deviations, indicative 
of long latencies, and very small interquartile deviations, indicative of 
short latencies) were shrunk the most. (See Table 1.) For example, again 
suppose a subject had an interquartile deviation of -.2 and a score of 1 for 
an item, and the mean item score was 3. His nondirectionally shrunken itf^m 
score would be 1 + 50% (3^1)- 2.0. 
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This measure used the same nine intervals as the directionally weighted 
item score, and its rates of shrinkage were in equal steps from 0% to 100^^ to 
0%. The rationale is also similar: the score is invalid for an item with 
either a very long or a very short latency, and hence the more extreme the 
latency is in either direction, the more the item's score is shrunk to the 
mean. 

This nondirectional measure is ad hoc, included on the basis of 
preliminary results with the double-standardized itom subgroup scores, which 
exhibited trends towards lower reusability and validity for scores with eithc" 
extremely short latencies or extremely long latencies. 

Insert Table 1 about here 



Deviant latency measures > Four measures of deviant latencies were also 
obtained (all excluded items for which the subject used the Help key): 

1. The product-moment correlation (transformed to Fisher's Z) between a 
subject's actual latencies and the samplers median actual latencies. This is 
an index of the correspondence between the subject's and the sample's 
latencies. 

2. The absolute difference between the subject's and the sample's 
median interquartile deviation for items. This is an index of the deviation 
between the subjec 's and the sample's average latencies. 

3. The subject's interquartile range for item interquartile deviations. 
This is an index of the variability of the subject's latencies. 
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4. The number of subject's double-standardized interquartile deviations 
of ?,5 or more. This is an index of outlying latencies (an interquartile 
deviation of 3.5 defines a "far out" outlier; Tukey, 1977).^ 
Other , Measures 

Several measures were include^', for exploratory purposes, as 
suppressor/moderator variables. Two were also used in screening the sample 
(Maximum Number of Changes Per Item and Effort on Test), attenuating their 
value as suppressor or moderator variables to some extent. 

Two measures were secured from the computer adminis^ration of the ASAP 
(both excluded items for which the subject used the Help key): 

1. Proportion of items changed. This is a control for individual 
differences in latencies associated with changes in response, 

2. Maximum number of changes per item. This? is an index of test-taking 
attitudes. 

Several measures were obtained from a paper-and-pencil questionnaire 
completed at the end of the testing session: 

1. Effort on test. This is the sum of the standard scores for three 
variables: 

a. At the beginning of the test, how hard did you try? Tried Very 
Littl- (1)... Tried Very Hard (4), 

b. At the end of the test, how hard did you try? Tried Very Little 
(1).,, Tried Very Hard (4). 

c. Overall, how hard did you try to do on the test? Tried Very 
Little (1),.. Tried Very Hard (4). 

2. Tiredness during test. This is the sum of the standard scores for 
two variables: 
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a* How tired did you foel at the beginning of the test? Extremely 
Tired (7) .Extremely Rested (1). 

b. Overall how tired did you feel at the end of the test? 
Extremely Tired (7) Extremely Rested (1). 

3. Computer use. This was derived from the following question: Within 
the ^ast year, how often have you used a computer? Never, or 1 to 10 times 
(1)...31 or More Times (4). 

In addition, the Armed Services Vocational Aptitude Battery (ASVAB; U.S. 
Department of Defense, 1984) Paragraph Comprehension scale score was obtained 
from the subjects* records. (The ASVAB was completed when the subjects applied 
for enlistment.) This is a 15-item r^asure of reading comprehension. 
Criterion 

The criterion was completion of six months (i.e., 180 days) of active 
service (or separation for "nonpejorative" reasons during that period: 
officer commission, breach of contract by the service, death, or early 
release), calculated from service entry date. This operational definition of 
retention is adapted from the one used in previous ASAP research (Trent et 
al., 1989). 
Procedures 

The ASAP, followed by one or more experimental cognitive tests, was 
computer adininistered to groups of approximately 30 subjects from February to 
May 1989. The questionnaire about test-taking attitudes and related matters 
was completed at the end of the session. The ASAP administration took 
approximately a half hour, and the entire administration about two and a half 
hours. The testing room held a battery of 34 personal computers. 
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The sample consisted of 1,090 Navy recruits (all men) at the Recruit 
Training Center (San Diego). 

All recruits it. the available units were asked to volunteer to 
participate in the study, but recruits who were not in the SAM or TAR programs 
(reservists with limited active-duty obligations) were given preference. (SAM 
and TAR recruits were not part of the study population.) The recruits were 
instructed that the test results would not affect their subsequent assignments 
or become part of their official records. 

The ASAP was administered to a total of 1,493 subjects. Forty- two 
subjects were eliminated because information was unavailable for most or all 
of their pertinent variables. An additional 136 subjects not part of the 
study population were excluded for one or more of these reasons: 

1. They were in the SAM program (no subjects were in the TAR program), 

2. They had prior military service (or information about this matter 
was missing) . 

3. They took the ASAP more than 15 days after service entry (or this 
information was missing) . 

4. They had a dominant language other than English.^ 

The remaining 225 subjects were eliminated because of their test-taking 
behavior or attitudes. This was done in two stages for the 1^315 subjects not 
already excluded. First, 122 subjects were dropped for either of two reasons: 

1. They used the Help key for more than one item. 

2. They reported on the paper-and-pencil questionnaire that they tried 
"Very Little" in the testing session (either at the beginning, at the end, or 
overall, or this information was missing). 
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Second, of the remaining subjects. 103 were eliminated for any of these 
reasons : 

1. They made more than five changes in their responses to an item. 

This corresponded to an interquartile deviation of 3.5 in the distribution for 
this variable — a "far out** outlier; 

2. They had a maximum double-standardized interquartile deviation of 
10,98 or more. This corresponded to an interquartile deviation of 3.5 in the 
distribution of maximum interquartile deviations — a "far out" outlier. 

3. They had a minimum actual latency of 2.21 seconds or less. This was 
the latency by the fastest .5% of the sample to the item with the shortest 
latencies, a criterion for improbably short latencies associated with 
premature responding (Jensen« 1985). 

Analyses 

Internal-consistency reliability was estimated by Coefficient Alpha for 
the regular ASAP score and the directionally and nondirectionally weighted 
item scores and by the intraclass correlation (Shrout & Fleiss, 1979, Case 1 
for mean ratings) for the item subgroup scores.. 

The product-^moment intercorrelations among the variables were computed. 
(Paragraph Comprehension scores were unavailable for 32 subjects, and the 
sample mean was substituted for the missing scores,) 

A series of regression analyses of the four kinds of ASAP measures was 
carried out against the retention criterion. The comparative validity of the 
measures was appraised from the zero^order correlations of the regular ASAP 
score, nondirectionally weighted item score, directi Mally weighted item 
score, and the multiple correlation of the item subgroup scores. 

Hi 



The incremental validity of the latency-based measures when combined 
with the regular ASAP score was assessed by hierarchical regression analyses 
(Cohen & Cohen, 1983): the zero-order correlation for the regular ASAP score 
was compared with the multiple correlation for the regular ASAP score plus the 
latency-based measure, '"his analysis was done separately for each latency- 
based measure. (In the analyses for the item subgroup scores, the scores were 
treated as a set, and Subgroup 10 was excluded to avoid collinearity between 
the regular ASAP score and the subgroup scores.) 

The ability of the deviant latency measures and the other variables to 
suppress or moderate the validity of the regular ASAP score and the latency- 
based ASAP measures was also assessed by hierarchical regression analyses A 
suppressor effect was evaluated by (a) a comparison of the zero-order or 
multiple correlation for the ASAP measure with the multiple correlation for 
the ASAP measure plus the suppressor/moderator variable and (b) a comparison 
of the corresponding Eero-order correlation and partial-regression weights for 
the suppressor/moderator variable, if the first comparison revealed a 
significant difference between the two correlations. (VHien suppression 
exists, the regression weight for a variable falls outside the boundaries set 
by its zero-order correlation and zero; Cohen & Cohen, 1983). These atuilyses 
were done separately for each ASAP measure. (In the analyses of item subgroup 
scores, the ten scores were treated as a set,) 

A moderator effect was evaluated by a comparison of the multiple 
correlation of the ASAP measure and the suppressor/moderator variable with the 
multiple correlation for the two variables plus their product term (the latter 
representing the interaction between the ASAP measure and the suppressor/ 
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moderator variable). In common with the suppressor analyses, these moderator 
analyses were done separately for each ASAP measure, and the item subgroup 
scores were treated as a set. 

Results 

Retention Criterion 

The retention rate was 91.2%: 994 subjects of the 1,090 subjects 
completed six months of active service (or separated for nonpejorative 
reasons),* comparable to the 91.3% retention rate for the same time period in 
previous ASAP research (T, Trent, personal communication, August 1986), 

The reasons for the 96 subjects' attrition, based on the Interservice 
Separation Codes r are reported in Table 2. The major reasons were fraudulent 
entry (28,1%), erroneous enlistment or induction (25,0%), and trainee 
discharge (21.9%). 

Given the extreme split in the retention criterion, the maximum product- 
moment correlation with it Is .57 (McNeraar, 1962). 



Insert Table 2 about here 



Reliability of ASAP Measures 

The internal-consistency reliability of the regular ASAP score and the 
latency-based ASAP measures is reported in Table 3. The reliability of the 
item subgroup scores is also shown in Figures 1 and 2. The regular ASAP score 
and the nondirectionally weighted item score were somewhat more reliable than 
the directionally weighted item score. The reliability was .80 for the 
regular ASAP score, comparable to the previously reported reliability of .77 
for a 125-item form (T. Trent, personal communication, December 1987). The 



21 



-17- 

rellability was also .80 for the directionally weighted item score (both the 
double-standardized and single-standardized versions), but the reliability was 
lower for the nondirectionally weighted item score: .74 for the double- 
standardized version and .72 for the single-standardized version. 

The reliability was also lower for the item subgroup scores: .14 to .31 
for the double-standardized versions, and .10 to .40 for the single- 
standardized versions. The trends for the two kinds of scores diverged 
markedly. For the double-standardized version, the reliability was noticeably 
lower for scores at both extremes (Subgroup 1, £^^-.14; 10, £^^-.23). For the 
single-standardized version, the reliability systematically increased from the 
score with the longest latencies (Subgroup 1, I„-.10) to the score with the 
shortest latencies (Subgroup 10, £^^-.40). 

Because the item subgroup scores were based on 12 items, a relevant 
comparison is the estimated reliability (using the Spearman-Brown formula) of 
.29 for the regular ASAP score with the same number of items. None of the 
double-standardized scores had appreciably highe- reliability whereas the two 
extreme scores had appreciably lower reliability (Subgroup 1, £^^-.14; 
Subgroup 10, £jj-.23). In contrast, the two single-standardized scores with 
the longest latencies had appreciably higher reliability than the .29 estimate 
(Subgroup 9, £„-.37; 10, £^^-.40), and the two scores with the shortest 
latencies plus a score with moderate latencies had noticeably lower 
reliability than this estimate (Subgroup 1, Xjt-.IO; 2, £„-.15; 5. 1^^- 21). 



Insert Table 3 and Figures 1 and 2 about here 
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Intercorrelations of ASAP Measures and Retention Criterion 

The intercorrelations of the ASAP measures and the retention criterion 
appear in Table 3. All the ASAP measures (double-standardized and single- 
standardized versions) were highly correlated. The regular ASAP score 
correlated .98 to .99 with the two versions of the directionally weighted item 
score and ,96 to .97 with the two versions of the nondirectionally weighted 
item score. The double-standardized versions of the directionally and 
nondirectionally weighted item scores correlated .95 with each other and the 
single-standardized versions correlated .93. And the multiple correlations 
were .99 and .97, respectively, between the double-standardized versions of 
the itera subgroup scores and the same versions of the directionally and 
nondirectionally weighted item scores; the corresponding correlations were 
1.00 and .99 for the single-standardized versions. 
Comparative Validity of ASAP Measures 

Figures 3 and 4 show the correlations of the ASAP item subgroup scores 
with the criterion. 

All the ASAP measures had the same level of vaHdlty. The regular ASAP 
score correlated .17 with the criterion^ comparable to the .18 to .20 
correlations with six-month retention reported previously (T. Trent, personal 
communication, August 1986). The correlations for directionally weighted and 
nondirectionally weighted item scores were similar: .16 for the two versions 
of the directionally weighted score, and *15 and .16 for the two versions of 
the nondirectionally weighted score. The multiple correlations of the item 
subgroup scores were also similar: .18 for both versions. 

The correlations for the Individual item subgroup scores were lower: 
.07 to .14 for the double-standardized version, and .07 to .12 for the single- 
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standardized version. Again, Che trends for the two versions diverged. For 
the double-standardized version, the trend was curvilinear: the correlation 
was appreciably higher for a middle score (Subgroup 6, £^.14) and noticeably 
lower for the scores at the extremes (especially Subgroup 10, £-.07). No 
trend was apparent for the single-^standardized version. 

A relevant comparison for these findings about the item subgroup scores 
is the estimated validity of .10 (using the Spearman-Brown formula) of the 
regular ASAP score for 12 items. Only one double-standardized score had 
appreciably different validity (Subgroi.p 6, £.-.14). None of the single- 
standardized scores had noticeably different validity than this estimate. 



Insert Figures 3 and 4 about here 



Incr_emen tal Validity of Latency-Based ASAP Measures 

The multiple regression analyses of the incremental validity of the 
latency-based ASAP measures are summarized in Table 4. None of the measures 
significantly (2>'05) increased the multiple correlation with the criterion 
when combined with the regular ASAP score. 



Insert Table 4 about here 



Incremental Validity of Suppressor/Moderator Variables 

The multiple regression analyses of the incremental validity of the 
suppressor/moderator variables are summarized in Table 5* None of these 
variables significantly (2>.05) increased the roultir e correlation with the 
criterion when combined with the regular ASAP score or with the latency-based 
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ASAP measures, indicating that the suppressor/moderator variables were not 
functioning as suppressor variables. 

With one exception, none of the variables significantly (e>.05) 
increased the multiple correlation when their product score was combined with 
the ASAP measure and the suppressor/moderator variable, indicating that the 
suppressor/moderator variables were not functioning as moderator variables. 
The exception involved the Interquartile Range and single-standardized item 
subgroup scores. The multiple correlation in this analysis increased to .2U 
from .19 when the product score was added (£<.05). 



Insert Table 5 about here 



Discussion 

Theoretical and M ethodological Implications 

It is apparent from the results that the latency-based ASAP measures did 
not improve the biographical inventory's predictive validity when used instead 
of or in addition to the conventional ASAP score. And it is equally clear 
that the measures of deviant latencies, along with the exploratory variables, 
did not function as suppressor or moderator variables to enhance the ASAP's 
validity either, (The single instance in which a deviant latency measure, 
displayed a moderator effect— the Interqu.^.tile Range vis-a-vis the single- 
standardized item subgroup scores— is not readily interpretable and probably 
represents a chance outcome of the large number of significance tests 
performed, ) 

Nonetheless, some important positive findings did emerge. Consistent 
with the expectation based on previous results that items with long latencies 
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are unstable, systematic trends in reliability occurred in the analysis of 
item subgroups with single-standardized latencies, with lower internal- 
consistency reliability for subgroups of items with longer latencies. The 
findings were much less clear cut in the reliability analysis of item 
subgroups with double-standardized latencies but suggested lower reliability 
for subgroups with either very long or very short latencies. This 
unanticipated possibility that items with unusually short latencies may also 
be unstable needs to be followed up. An obvious conjecture is that very short 
latencies indicate subjects are paying minimal attention to the item content 
or, at worst, are responding more-or-less randomly* The sample was screened 
to eliminate subjects with poor test-taking attitudes, including individuals 
making impossibly fast responses, but this process excluded only those with 
extreme behavior. 

The trends in reliability in this analysis of single-standardized 
latencies support and extend itemmetric studies that uncovered a substantial 
association between latencies and Instability for personality items (Dunn et 
al., 1972; Holden et al,, 1988; Holden & Fekken, 1990), Because the single- 
standardization procedure was used, considerable commonality probably exists 
in ^he items that make up the item subgroup scores. The present findings 
indicate that the earlier results about retest reliability also apply to 
internal-consistency reliability and suggest that the previous findings were 
not simply an artifact of the dichotomous character of the personality items 
(Goldberg, 1963). The failure of these clear-cut trends in reliability to be 
paralleled by similar trends in validity may occur because of the generally 
low level of validity involved. 

o 
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These reliability trends have iciplications for writing and selecting 
Inventory items. First, reliability (and, in turn, validity) may be maximized 
by writing items that elicit short latencies • Such items are primarily short 
in length and unambiguous (Dunn et al., 1972; Holden et al , , 1988). 

Second, latency data may be useful in selecting items for reliability 
(and, indirectly, for validity) (Fekken & skson, 1988; Holden & Jackson, 
1990). Standard item analytic methods that choose items with high 
correlations with the total score or the criterion can accomplish these 
purposes, too. But latency data may be particularly useful when (a) the 
measure is heterogeneous and hence item-total score correlations are of 
questionable value, (b) the criterion has limited validity, or (c) the 
criterion requires time to mature. 

Another important finding concerns the expectation that items with long 
latencies are less valid. The findings in the analysis of double-standardized 

s'»bgroup scores sug;^ested that items with very long latencies as well as 
those with very short larencies were less valid. Furthermore, this analysis 
identified a subset of items with moderate latencies (Sibgroup 6) that were 
more valid than the othui: spts and almost as v .d as the regular ASAP score. 
Indeed, the estimated validity of this subgroup score would be .24 (using tlie 
Spearman-Brown formula) if it had as many items as the regular ASAP score, 
appreciably larger than the latter 's validity of .17. This resign clearly 
needs to be replicated, but it offers the intriguing prospect of improving the 
ASAP*s validity by using more of the same kind of items that are in this 
subgroup* Because the double-standardization procedure clustered items on the 
basis of their subject by item interactions, it is unlikely that appreciable 
commonality exists m the items in this subgroup. Consequently, it would 
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probably be necessary to identify the appropriate items individually for each 
subject, using computerized adaptive testing. How accurately such items can 
be identified remains to be seen. 

One other ou^como is noteworthy. The similar validity of the ASAP 
regular score and the latency-based measures indicates that the unorthodox 
methods used to devise the latter— the Item subgroup scores that rely on 
comparable &*cores from different sets of heterogeneous items, and the 
weighting scheme used in the directionally and nondirectionally weighted item 
scores — did not degradi* the ASAP's validity. This outcome implies that these 
unusual procedures were reasonable. 

All in all, the findings for the item subgroup scores, as well as the 
results for the weighted item scores, offer mixed support for the two kinds of 
competing conceptualizations: (a) Individual differences by item 
characteristics interaction, represented by the doublets tandardize J measures; 
and Co) item characteristics, represented by the single-standardized measures. 
The roost clear-cut confirmation was associated with the reliability findings 
for the single-standardized item subgroup scores. 

It should be recognized in this connection that the empirical keying of 
the ASAP items hampered the ability of the single-standardization procedure to 
improve validity. Insofar as the subjects' item latencies are in the same 
order, and the present sample is comparable to the one used in deriving the 
item scores, latency data will not improve validity because the items already 
have optimal weights for predicting the criterion. For instance, suppose that 
the iteias with the longest latencies for everyone in the sample were also the 
least valid. The weights for the items reflect the level of validity for the 
sample^ and adjusting the weights ir the same way for each subject (because 
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all subjects have the same latencies) will have no effect. The ASAP's 
empirical keying does not affect the reliability results for the single- 
standardized procedure because the items* weights are not optimal for 
reliability. The keying also does not affect either the validity or 
reliribility results for the double-standardized procedure because this 
standardization makes the latencies and the resulting adjustments different 
for each subject. 

The findings also offer no basis for choosing between the dlrectionally 
and nondlrectionally weighted items for the two performed similarly: their 
validity was equivalent ^ though the directional version was somewhat more 
reliable, 
Future Directions 

Future efforts might benefit by partitioning response latency into its 
major components. Rogers (1974a; 1974b, p. 130) has distinguished three main 
stages in responding to an inventory item: stimulus encoding, stimulus 
comprehension, and binary "true/false" decision about the item. This last 
stage, in turn, is divided into two substages: self-referent decision 
("relating <^f the internalized item content to the 'self-concept'") and 
response selection. Similarly, Kuncel (1973, p. 547) has delineated two 
stages: meaning ( 'attributing some 'reasonable' interpretation to the item") 
and apply ("employing information which is * /?ell-suited' as a basis for 
answering the item")* 

The present study, In common with most research in this field, measured 
and used the total time involved in all these stages of responding, but the 
time consumed in the various stages may have very different psychological 
implications. The conceptualizations linked with item characteristics (e,g., 
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Dunn et al, , 1972; Hanley^ 1962) focus on the early stages (e.g, » Rogers' 
stimulus encoding and stimulus comprehension; Kuncel's cifianing) . In contrast, 
conceptualizations concerned with the interaction between individual 
differences and item characteristics (e.g,, Gilbert, 1967; Kuncel, 1973; 
Markus, 1977; Rogers, 1974a, 1974b; Teraplu & Geisinger, 1990) primarily deal 
with the later stages \|Rogers' binary decision stage, Kuncel^s apply stage) 

As a first step in partitioning latencies, it would be prudent to 
control experimentally for item length, as had been done in a few studies 
(Kanley, 1962; Rogers, 1974a, 1974b; Temple & Ge; inger, 1990), because of its 
generally high associati'^n with latencies (Dunn et al, , 1972; Holden et al., 
1985; Tetrick, 1989). 

It would be worthwhile to employ a more predictable criterion, as 
sugges :ed earlier. The ASAP's validity was modest against the six-month 
retention criterion in this study, with correlations accounting for less than 
4% of the criterion variance. In these circumstances, substantial increases 
in validity with improvements in the ASAP are difficult to uncover, even when 
reliability is dramatically enhanced, as in the analysis of the single- 
standardized item subgrou;^ scores. At the least, a 36-month retention 
criterion might be used, given the substantially higher validity of the ASAP 
with 36-raonth retention (£-.27; Trent, 1989) than with 6-month retention 
(£-.18 to .20; T. Trent, personal communication, August 1986). Other 
criteria for the ASAP might also be employed, such as disciplinary records, 
promotions, and recommendations for re-enlistment. 

It might also be fruitful to investigate situational influences. 
Latencies may convey more significant information, and consequently latency-- 
based measures may be more useful, in situations that are perceived as more 
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important than the typical research study. Investigations that use the 
measures in actual selection or in faking experiments that approximate such 
demand characteristics are especially relevant. It is noteworthy that a 
recent faking experiment (Hsu, Santelli, & Hsu, 1989) found that latency 
measures (the mean latencies for subtle and obvious MMPI items) were able to 
detect dissimulation. 

More research is obviously needed on methods for weighting individual 
items and for grouping items into subsets by their latencies. The schemes 
used in this study were no more than first approximations, and a variety of 
improvements are possible. 

Finally, a systematic appraisal of the efficacy of latency data in 
writing and selecting inventory items is called for* Studies that 
experimentally manipulate item latencies, via changes in the length and other 
characteristics of the items, and then compare the items' reliability and 
validity are pertinent. Equally relevant are investigations that assess the 
relative reliability and validity of items selected on the basis of latency 
data with items chosen by standard item analytic methods. 
Conclusion 

Although this initial effort at using response latency data to improve 
the validity of a biographical inventory directly was unsuccessful, there were 
strong indications that employing these data in developing an inventory may 
enhance validity indirectly, and thereby accomplish the same goal. 

It should also be borne in mind that closely related work has directly 
improved the validity of personality inventories- Several recent studies have 
found that latency scores for a scale (i.e., the mean latency for endorsed 
items on the scale, and the mean latency for rejected items on the scale) 
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frequently had incremental validity in predicting external criteria, when 
combined with the regular scale score (Fekken, 1990; Holden, Fekken, & Cotton, 
in press; Mervieldc. 1988; Popham & Holden, 1990). This particular approach 
requires items that are dichotomous and homogeneous , and hence is inapplicable 
to a heterogeneous biographical inventory, such as the ASAP. Nonetheless, 
these findings underscore the potential for latency data. 

Given the ease of collecting response latency information, its ability 
to improve the validity and utility of self-report inventories merits serious 
investigation. 
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Footnotes 

^The signs of the correlations in the Dunn et al. and the Holden and 
Fekken studies have been reflected to be consistent with the reversal of the 
dependent variable in these investigations from the proportion of unchanged 
responses to the proportion of changed responses. 

^Several of the current weights were assigned on rational grounds to 
improve content validity; it was not feasible to replicate that process for 
this study. 

^An interquartile deviation of 2.00 defines the "inner fence" of a 
frequency distribution (Tukey, 1977). 

^An interquartile deviation of 3.50 defines the "outer fence" of a 
frequency distribution (Tukey, 1977). 

^Dominant language was assessed by the following question that was 
computer administered, immediately preceding the ASAP: What language do you 
read and write best? (A) English. (B) Spanish, <C) Chinese, (D) Tagalog, (E) 
Some ot)ier language. 

^wo subjects were separated for nonpejorative reasons: Non-Battle 
Death — Other, and Death — Cause Not Specified* 
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Table I 

Shrinkage Rates for Dlrectlonallv and Nondtrectl onallv WeiRhted.Itgm Scores 



Interquartile Deviation 



Shrinkage Rate 



Directional Nondirectionnl 



-.81 


or 


more 


0.0% 


100.0% 


-.80 


to 


-.41 


12.5 


75.0 


-.40 


to 


-.01 


25.0 


50,0 


.00 


to 


.39 


37.5 


25.0 


.40 


to 


.79 


50.0 


0.0 


.80 


to 


1.19 


62.5 


25.0 


1.20 


to 


1.59 


75.0 


50.0 


1.60 


to 


1.99 


87.5 


75.0 


2.00 


or 


More 


100.0% 


100.0% 
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Table 2 




lleasons fp^- Attpitloyi. N - 96 




Reason* 


Percent 




3 1% 


MpdtrAllv unfliiAllfied fnr active dutv—— other 


4.2 




8 3 


Drugs 


1 0 


r i. auuv* 1611 V 6iiui»y 


OR 1 


uUUIJ vl. wilt? os^LyfJLL.^ 


2 . 1 




2. 1 


Trainee discharge 


21.9 


Erroneous enlistment or induction 


25.0 


Other 


4.2% 



Interservice Separation Code 



40 



Tftblt 3 



lmerCprreieU.tyr» RehtbHUy of Segu^ir and L a tency-Based ASAP Mgnurts and thg Rgteotioo Cr^terior t| 



Neature 



Mean 



S.D. 



1. 


Re^wlar ASAP 


274.t3 


16.32 


2. 


DirectiooaUy Wefshted 








itm Scofft'-D.s. 


2.30 


.09 


3, 


tondirectfonaUy weighted 


2.23 


.07 




Han S<;ore'-O.S. 






4. 


Mem Subgrixp Score l-D.S. 


2.32 


.22 


5. 


ixm Subgroup Score 2-'D,S. 


2.26 


.24 


6. 


\tm Subgroup Score ^'-D-S. 


2.25 


.23 


7, 


Itea Subgroup Score 4"0.S. 


2.25 


.24 




Ite« Subgroup Score 5--D.S. 


2.33 


.23 


9. 


Ite« Si^roiip Score 6''0,S. 


2.3' 


.24 


10. 


Ite«) Subgrovp Score 7--0.S. 




.23 


11. 


Iteffi Subgroup Score 8--D.S. 


2.25 


.24 


12, 


Itefli Subgroup Score 9 -O.S. 


2.27 


,24 


13. 


Item Subgroup Score 10--D.S. 


2.i3 


.24 


U« Of rectionally Vefohtcd 








fxtm Score- -s.S. 


2,29 


.09 


15. 


Nondfrectionalty Weighted 






16. 


Itan Score- -S.S. 


2. a 


.07 


Item Subgroup Score 1--S.$. 


z.zr 


.21 


17. 


\Xem Subgroup Score 2-S.S. 


2.25 


.22 


18. 


\X9m Subgro44> Score 3--S.S. 


2.31 


.23 


19. 


lX9m Subgroup Score 4-*S.S. 


2.25 


.23 


ZO. 


Mem SuDgroup Score 5--S.<. 


2. a 


.23 


21, 


Iteti $i4»groip Score 6"S.S. 


2.2h 


.24 


22. 


Ixtm Subgroup Score 7"S.S. 


2.2% 


.24 




item Subgroup Score 8 -S.S. 


2.30 


.25 


24. 


Ite« Subgroup Score 9--S.S. 


2.3- 


.26 


25. 


Ite« St^group Score 10- S.S, 


2,».5 


.26 


26- 


Retention Criterion 




.28 



10 



11 



12 13 



14 



15 



(80) 99 97 



£74) 



51 


58 


61 


55 


58 


59 


60 


61 


58 


55 




51 


57 


60 


54 


5* 


56 


59 


60 


57 


53 


99 


50 


56 


60 


54 


55 




59 


59 


55 


53 


95 


£14) 


19 


28 


20 


23 


23 


22 


23 


24 


17 


50 




(27) 


26 


21 


26 


26 


32 


27 


25 


27 


57 






(26) 


28 


28 


27 


31 


30 


28 


27 


61 








(27) 


30 


25 


24 


23 


24 


19 


54 










(25) 


23 


24 


30 


25 


26 


57 












(2S) 


27 


31 


25 


25 


56 














(26) 


3? 


28 


25 


58 
















<3i) 


28 


26 


60 


















(26) 


23 


56 




















£23) 


53 






















(80) 



94 
95 

49 

55 
58 
51 
57 
57 
59 
59 
55 
54 



16 


17 


18 


19 


20 


21 


22 


23 


24 


25 


26 


LX 


?v 


c« 


>o 


CA 


5y 


58 


62 


66 


67 


17 


jO 


LL 


>u 




PV 


60 


58 


64 


68 


69 


16 


Lit 




>Ji 




50 


CA 

59 


55 


58 


61 


61 


IS 




£3 




CO 






29 


32 


36 


35 


08 


22 


7ft 
£0 




J** 




3> 




36 


38 


43 


07 


23 








3? 


XT 


33 


36 


40 


43 


09 


25 


23 






1 

y* 


5!> 


XT 


33 


36 


38 


08 


22 


26 


30 






3^ 


XT 

if 


ic 

3) 


3o 


38 


10 


24 


29 


29 


32 


35 


36 


31 


39 


41 


37 


14 


CO 


30 


34 


33 


3t 


35 


35 


39 


58 


35 


10 


25 


30 


35 


32 


35 


39 


33 


38 


39 


44 


09 


29 


25 


33 


31 


32 


31 


33 


32 


42 


39 


11 


28 


30 


32 


32 




50 


32 


34 


34 






31 


42 


48 


54 


57 


60 


59 


65 


70 


71 


16 


37 


52 


61 


61 


6'. 


57 


56 


56 


57 


55 


16 


(101 


13 


21 


20 


20 


17 


13 


14 


21 


20 


07 




(15) 


15 


22 


1? 


20 


20 


27 • 


19 


25 


11 






(24) 


18 


Z\ 


25 


25 


24 


27 


26 


09 








(25) 


2B 


23 


28 


23 


30 


51 


07 












30 


25 


28 


31 


35 


10 












(25) 


27 


31 


33 


31 


07 














(26) 


29 


30 


32 


13 
















(32) 


38 


56 


08 


















(37) 


43 


10 




















(40) 


12 






















(•-) 


.05 and 


.01 Uyei» 


(two-tail). 


respect Wety. 


ReUebiiity 







fiats. Decfnist points have been omittcc <or correUtlon* tncf reli»bilily eo*(f ieieoti. correl*tions o* .06 .nd .08 ire sisnificant at 
co.ffici«,t« .ppe-r in prentheses. (*,*ai>.Uty e.t,».ted by Co*ftieient Alph. for the Res^lar ASV. Dl rect ,on.t ly weishted ,t«. Score. ^ Mo^i rect ,00,11 y weisr-t^d Ite^ Score, orxl by the intr»ct.« 
correlation for the ItefD Subgroup Sco'es. 



f 

as 
I 



41 



42 
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Table 4 



Incremental Valid ity of Latencv-Based ASAP Measures when Combined with Regular 
ASAP Measures. N - 1.090 



Measures 



£ 



g Increase 



Regular ASAP 

Regular ASAP and Directionally 
Weighted Item Score — Double 
Standardized 



1653** 



1681** 



0028 



Regular ASAP 



Regular ASAP and Nondirectionally 
Weighted Item Score — Double 
Standardized 



1653** 



1667** 



0014 



Regular ASAP 

Regular ASAP and Item Subgroup 
Scores — Double Standardized 



10 



1653** 



1809** 



0156 



Regular ASAP 

Regular ASAP and Directionally 
Weighted Item Score — Single 
Standardized 



1653** 



1656** 



0003 



Regular ASAP 

Regular ASAP and Nondirectionally 
Weighted Item Score — Single 
Standardized 



1653** 



1660** 



.0007 



Regular ASAP 

Regular ASAP and item Subgroup 
Scores — Single Standardized 



10 



1653** 



1826** 



0173 



ERIC 



**£ <.01 
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Table 5 



Incremental Validity of Suppressor/Moderator Variables when Combined with 
Regular and Latencv-Based ASAP Measures. N - 1.090 



Measures 



E 



R Increase 



Regular ASAP 

Regular ASAP and Correlation Between 
Subject's and Sample's Latencies 

Regular ASAP, Correlation Between 
Subject's and Sample's Latencies, 
and Product 



.1653** 



.1709** 



1709** 



.0056 



0000 



Regular ASAP 

Regular ASAP and Interquartile Range 

Regular ASAP, Interquartile Range, 
and Product 



X 
2 



1653** 
1728** 

1728** 



0075 



0000 



Regular ASAP 

Regular ASAP and Interquartile 
Deviations of 3.5 or More 

Regular ASAP, Interquartile 

Deviations of 3.5 or More, and 
Product 



1653** 



1667** 



1713** 



0014 



0046 



Regular ASAP 

Regular ASAP and Difference Between 
Median Interquartile Deviation 
for Subject and Sample 

Regular ASAP, Difference Between 
Median Interquartile Deviation 
for Subject and Sample, and Product 



1653** 



1656** 



1662** 



,0003 



,0006 



ERIC 
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Measures 



E 



R Increase 



Regular ASAP 

Regular ASAP and Items Changed 

Regular ASAP, Items Changed, and 
Product 



1 

2 



1653** 

1669** .0016 



1731** 



.0062 



Regular ASAP 

Regular ASAP and Maxiraum Changes 
Per Item 

Regular ASAP, Maximum Changes Per 
Item, and Product 



1653** 



1728** 



1728** 



0075 



0000 



Regular ASAP 

Regular ASAP and Effort on Test 

Regular ASAP. Effort on Test, and 
Product 



Regular ASAP 

Regular ASAP and Tiredness During 
Test 

Regular ASAP, Tiredness During 
Test, and Product 



Regul.ir ASAP 

Regular ASAP and Computer Use 

Regular ASAP, Computer Use, and 
Product 



1 

2 



1 

2 



1653** 
1690** 

1730** 



1653** 



1727** 



1773** 



1653** 
1654** 

1674** 



,0037 



0040 



0074 



0046 



0001 



0020 



ERIC 
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Table 5 (Continued) 



Measures 



Regular ASAP 

Regular ASAP and Paragraph 
Comprehension 

Regular ASAP, Paragraph 
Comprehension and Product 



Directionally Weighted Item 
Score — Double Standardized 

Directionally Weighted Item 
Score and Correlation Between 
Subject's and Samplers 
Latencies 

Directionally Weighted Item 
Score, Correlation Between 
Subject's and Sample's Latencie^p, 
and Product 



Directionally Weighted Item 
Score — Double Standardized 

Directionally Weighted Item 
Score and Interquartile Range 

Directionally Weighted Item 

Score, Interquartile Range, and 
Product 



Directionally Weighted Item 
Score — Double Standardized 

Directionally Weighted Item 
Score and Interquartile 
Deviations of 3,5 or More 

Directionally Weighted Item 

Score, Interquartile Deviations 
of 3.5 or More, and Product 



2 E B Increase 

1 .1653** 

2 .1688** .0035 

3 .1707** .0019 

1 .1581** 

2 .1641** .0060 

3 .1642** .0001 

1 .1581** 

2 .1662** ,0081 

3 .1662** .0000 

1 .1581** 

2 .1595** .0014 

3 .1619** ,0024 



ERLC 
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Table 5 (Continued) 



Measures 


Q 


E 


g Increase 


Directionally Weighted Item 
Score — Double Standardized 


1 


.1581** 




Directionally Weighted Item 
Score and Difference Between 
Median Interquartile Deviation 
for Subject and Sample 


2 


1583** 


000? 


Directionally Weighted Item 
Score, Difference Between 
Median Interquartile Deviation 
for Subject and Sample, and 
Product 


3 


. 1602** 


.0019 


Directionally Weighted Item 
Score — Double Standardized 


1 


.1581** 




Directionally Weighted Item 
Score and Items Changed 


2 


.1596** 


.0015 


Directionally Weighted Item 
Score, Items Changed ^ and 
Product 


3 


.1647** 


.0051 


Directionally Weighted Item 
Score — Double Standardized 


1 


. 1581** 




Directionally Weighted Item 
Score and Maximum Changes 
Per Item 


2 


.1658** 


.0077 


Directionally Weighted Item 
Score, Maximum Changes Per 
Item, and Product 


3 


.1659** 


.0001 


Directionally Weighted Item 
Score — Double Standardized 


1 


.1581** 




Directionally Weighted Item 
Score and Effort on Test 


2 


.1623** 


.0042 


Directionally Weighted Item 
Score, Effort on Test, and 
Product 


3 


.1659** 


.0036 
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Table 5 (Continued) 



Measures 



E 



g Increase 



Directionally Weighted Item 
Score — Double Standardized 



1581** 



Directionally Weighted Item 
Scox"e and Tiredness During 
Test 



1660** 



.0079 



Directionally Weighted Item 
Score and Tiredness During 
Test, and Product 



1700** 



.0040 



Directionally Weighted Item 
Score — Double Standardized 



ISBIhA: 



Directionally Weighted Item 
Score and Compucer Use 



1581** 



0000 



Directionally Weighted Item 
Score, Computer Use, and 
Product 



1621** 



.0040 



Nondirectionally Weighted Item 
Score — Double Standardized 



1547** 



Nondirectionally Weighted Item 
Score and Correlation Between 
Subject's and Sample's Latencies 

Nondirectionally Weighted Item 
Score, Correlation Between 
Subject's and Samplers Latencies, 
and Pi'oduct 



1619** 



.1620** 



,0072 



.0001 



Nondirectionally Weighted Item 
Score — Double Standardized 



1547** 



Nondirectionally Weighted Item 
Score and Interquartile Range 

Nondirectionally Weighted Item 
Score, Interquartile Rangt^ and 
Product 



,1638** 



.1640** 



,0091 



.0002 



ERIC 
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Table 5 (Continued) 



Measures 


n 


a 


B Increase 


Nondirectionally Weighted Item 
Score — Double Standardized 


1 


. 1547** 




Nondirectionally Weighted Item 
Score and Interquartile 
Deviations of 3*5 or More 


2 


. 1564** 


.0017 


Nondirectionally Weighted Item 
Score , Interquartile 
Deviations of 3.5 or More, 
and Product 


3 


.1636** 


.0072 


Nondirectionally Weighted Item 
Score — Double Standardized 


1 


.1547** 




Nondirectionally Weighted Item 
Score and Difference Between 
Median Interquartile Deviation 
for Subject and Sample 


2 


.1550** 


.0003 


Nondirectionally Weighted Item 
Score, Difference Between 
Median Interquartile Deviation 
for Subject and Sample, and Product 


S 


.1551** 


.0001 


Nondirectionally Weighted Item 
Score — Double Standardized 


1 


.1547** 




Nondirectionally Weighted Item 
Score and Items Changed 


2 


. 1564** 


.0017 


Nondirectionally Weighted Item 
Score, Items Changed, and Product 


3 


. 1607** 


.0043 


Nondirectionally Weighted Item 
Score — Double Standardized 


1 


.1547** 




Nondirectionally Weighted Item 
Score and Maximum Changes Per 
Item 


2 


.1627** 


.008J 


Nondirectionally Weighted Item 
Score, Maximum Changes Per 
I tern » and Product 


3 


.1629** 


.0002 
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Measures 



E 



S Increase 



Nondirectionally Weighted Item 
Score — Double Standardized 

Nondirectionally Weighted Item 
Score and Effort on Test 



,1547** 



1585** 



.0038 



Nondirectionally Weighted Item 
Score, Effort on Test, and 
Product 



1603** 



0018 



Nondirectionally Weighted Item 
Score — Double Standardized 

Nondirectionally Weighted Item 
Score and Tiredness During Test 

Nondirectiona** ly Weighted Item 
Score, Tiredness During Test, 
and Product 



1547** 
1629** 

1698** 



0082 



,0069 



Nondirectionally Weighted Item 
Score — Double Standardized 



1547** 



Nondirectionally Weighted Item 
Score and Computer Use 

Nondirectionally Weighted Item 
Score, Computer Use, and 
Product 



1547** 



1572** 



0000 



.0025 



Nondirectionally Weighted Item 
Score — Double Standardized 

Nondirectonally Weighted Item 

Score and Paragraph Comprehension 

Nondirectionally Weighted Item 
Score, Paragraph Comprehension, 
and Product 



1547** 



1582** 



1596** 



0033 



.0014 
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Measures U S S Increase 



Item Subgroup Scores — Double 
Standardized 

Item Subgroup Scores and 

Correlation Between Subject's 
and ample 's Latencies 

Item Subgroup Scores, Correlation 
Between Subject's and Sample's 
Latencies, and Product 



10 ,1810** 



11 .1870** .0060 



21 .2037** .0167 



Item Subgroup Scores — Double 

Standardized 10 .1810** 

Item Subgroup Scores and 

Interquartile Range 11 .1887** .0077 

Item Subgroup Scores, 
Interquartile Range, 

and Product 21 .2079** .0192 



Item Subgroup Scores — Double 

Standardized 10 .1810** 



Item Subgroup Scores and 

Interquartile Deviations of 
3 . 5 or More 

Item Subgroup Scores, Interquartile 
Deviations of 3.5 or More, and 
Product 



11 .1820** .0010 



21 .2079** .0259 



ERIC 



Item Subgroup Scores — Double 

Standardized 10 .1810** 

Item Subgroup Scores and 
Difference Between Median 
Interquartile Deviation for 

Subject and Sample 11 ,1812** .0002 

Item Subgroup Scores, Difference 
Between Median Interquartile 
Deviation for Subject and 

Sample, and Product 21 .1992** .0180 
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Measures 



E 



E Increase 



Item Subgroup Scores — Double 
Standardized 



10 



.1810** 



Item Subgroup Scores and Items 
Changed 

Item Subgroup Scores, Items 
Changed, and Product 



11 



21 



Item Subgroup Scores--Double 
Standardized 



10 



1826** 



2087** 



1810** 



,0016 



,0261 



Item Subgroup Scores and 
Maximum Changes Per Item 

Item Subgroup Scores, Maximum 
Changes Per Item, and Product 



11 



21 



.1877** 
. 1971** 



,0067 



,0094 



Item Subgroup Scores — Double 
Standardized 



10 



1810** 



Item Subgroup Scores and Effort 
on Test 



11 



1850** 



0040 



Item Subgroup Scores, Effort on 
Test, and Product 



21 



.2051** 



,0201 



Item Subgroup Scores — Double 
Standardized 



10 



1810** 



Item Subgroup Scores and 
Tiredness During Test 

Item Subgroup Scores, Tiredness 
During Test, and Product 



11 



21 



1870** 



2111** 



,0060 



,0241 



Item Subgroup Scores — Double 
Standardized 



10 



1810** 



Item Subgroup Scores and 
Computer Use 

Item Subgroup Scores, Computer 
Use, and Product 



11 



21 



1010** 



1937** 



,0000 



0127 
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Tablo 5 (Continued) 



Measures 



n 



& 



E Increase 



Item Subgroup Scores — Double 
Standardized 



10 



1810*^ 



Item Subgroup Scores and 
Paragraph Corrprehension 

Item Subgroup Scores, Paragraph 
Comprehension, and Product 



11 



21 



1842** 



1892** 



.0032 



.0050 



Directionally Weighted Item 
Score — Single Standardized 

Directionally Weighted Item 
Score and Correlation Between 
Subject's and Sample's Latencies 

Directionally Weighted Item 
Score, Correlation Between 
Subject •s and Sample's 
Latencies, and Product 



1609** 



1670** 



.0061 



1671** 



.0001 



Directionally Weighted Item 
Score — Single Standardized 

Directionally Weighted Item 
Score and Interquartile Range 

Directionally Weighted Item 
Score, Interquartile Range, 
and Product 



1609** 



1691** 



1691** 



.0082 



,0000 



Directionally Weighted Item 
Score — Single Standardized 

Directionally Weighted Item 
Score and Interquartile 
Deviations of 3.5 or More 



1609** 



1624** 



,0013 



Directionally Weighted Item 
Score , Interquartile 
Deviations of 3.5 or More, 
and Product 



•1651** 



,C027 
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Measures 


Q 


R 


R Increase 


Directlonally Weighted Item 
Score — Single Standaradized 


1 


.1609** 




Directionally Weighted Item 
Score and Difference Between 
Median Interquartile Deviation 
for Subject and Sample 


2 


.1612** 


.0003 


Directionally Weighted Item 

Score, Difference Between Median 
Interqt:artile Deviation for 
Subject and Sample, and Product 


3 


,1629** 


.0017 


Directionally Weighted Item 
Score — Single Standardized 


1 


.1609** 




Directionally Weighted Item 
Score and Items Changed 


2 


. 162^** 


.0015 


Directionally Weighted Item 
Score, Items Changed, and 
Product 


3 


. 1697** 


.0073 


Directionally Weighted Item 
Scores — Single Standardized 


I 


.1609** 




Directionally Weighted Item 
Score and Maximum Changes 
Per Item 


2 


.1685** 


.0076 


Directionally Weighted Item 
Score, Maximum Changes Per 
Item, and Product 


3 


. 1685** 


.0000 


Directionally Weighted Item 
Score — Single Standardized 


I 


. 16C?** 




Directionally Weighted Item 
Score and Effort on Test 


2 


.1650** 


.0041 


Directionally Weighted Item 
Score, Effort on Test, and 
Product 


3 


.1682** 


.0032 
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Table 5 (Continued) 



Measures 


0 


£ 


£ Increase 


Directionally '^sighted Item 
Score — Single Standardized 


1 


.I609*-v 




Directionally Weighted Item 

Score and Tiredness During Test 


2 


.1686** 


.0077 


Directionally Weighted Item 
Score, Tiredness During Test, 
and Product 


3 


.1732** 


.0046 


Directionally Weighted Item 
Score — Single Standardized 


1 


. 1609** 




Directionally Weighted Item 
Score and Computer Use 


2 


. 1610** 


.0001 


Directionally Weighted Item 
Score, Computer Use, and 
Product 


3 


. 1659** 


. 00A9 


Nondirectionally Weighted Item 
Score — Single Stand£^rdi2ed 


1 


.1630** 




Nondirectionally Weighted Item 
Score and Correlation Between 
Subject's and Sample's Latencies 


2 


. 1699** 


.0069 


Nondirectionally Weighted Item 
Score, Correlation Between 
Subject s and Sample s 
Latencies, and Product 


3 


.1707** 


.0008 


Nondirectionally Weighted Item 
Score — Single Standardized 


1 


.1630** 




Nondirectionally Weighted Item 
Score and Interquartile Rang^ 


2 


.1711** 


.0081 


Nondirectionally Weighted Item 
Score, Interquartile Range, 
and Product 


3 


.1711** 


.0000 
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Table 5 (Continued) 



Measures 


Q 




R T nr f a ji <;p 


Nondirectionally Weighted Item 
Score — Single Standardized 


1 


,1630** 




Nondirectionally Weighted Item 
Score and Interquartile 
Deviations of 3.5 or More 




» iOH O 


0016 
* \j \j ^ \j 


Nondirectionally Weighted Item 
Score, Interquartile Deviations 
of 3.5 or More, and Product 


3 


.1708** 


.0062 


Nondirectionally Weighted Item 
Score — Single Standardized 


1 


.1630** 




Nondirectionally Weighted Item 
Score and Difference Between 
Median Interquartile Deviation 
for Subject and Sample 


2 


.1633*- 


.0003 


Nondirectionally Weighted Item 
Score, Difference Between 
Median Interquartile Deviation 
for Subject and Sample, and 
Product 


3 


.1643** 


.0010 


Nondirectionally Weighted Item 
Score — Single Standardized 


1 


.1630** 




Nondirectionally Weighted Ite.n 
Score and Items Changed 


£■ 


. I DM / « ^ 


0017 


Nondirectionally Weighted It=^m 
Score, Items Changed, and 
Product 


O 


1 7 1 7** 

. 1 / A. / ^ " 


0070 


Nondirectionally Weighted Item 
Score — Single Standardized 


1 


.1630** 




Nondirectionally Weighted Item 
Score and Muximum Changes 
Per Item 


2 


.1704** 


.0074 


Nondirectionally Weighted Item 
Score, Maximum Changes Per 
Item, and Product 


3 


.1706** 


.0002 
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Table 5 (Continued) 



Measures 


Q 


E 


g Increase 


Nondirectlonally Weighted Item 
Score — Single Standardized 


1 


.1630** 




Nondirectlonally Weighted Item 
Score and Effort on Test 


2 


, lob/** 




Nondirectlonally Weighted Item 
Score, Effort on Test, and 
Product 


3 


. 1704** 


, 0037 


Nondirectlonally Weighted Item 
Score — Single Standardized 


1 


. 1630** 




Nondirectlonally Weighted Item 
Score and Tiredness During 
Test 


2 


. 1703** 


. 0073 


Nondirectlonally Weighted Item 
Score, Tiredness During Test, 
and Product 


3 


.1747** 


.0044 


Nondirectlonally Weighted Item 
Score — Single Standardized 


1 


. 1630** 




Nondirectlonally Weighted Item 
Score and Computer Use 


I 


. 1630** 


. UUUU 


Nondirectlonally Weighted Item 
Dcore , uompuuer use , ano 
Product 


3 


. 1635** 


.0003 


Nondirectlonally Weighted Item 
Score — Single Standardized 


1 


.1630** 




Nondirectlonally Weighted Item 

Score and Paragraph Comprehension 


2 


. 1669** 


.0039 


Nondirectlonally Weighted Item 
Score, Paragraph Comprehension, 
and Product 


3 


. 1680** 


.001 1 
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Table 5 (Continued) 



Measures 


n 
ii 


R 


R Increase 


Item Subgroup Scores — Single 
Standardized 


10 


.1813** 




Item Subgroup Scores and 

Correlation Between Subject's 
and Sample's Latencies 




. 1867** 


.0054 


Item Subgroup Scores, Correlation 
Dewween iUDject s aiiu ^c^uipic? » 
Latencies, and Product 


21 


.2094** 


.0227 


Item Subgroup Scores — Single 
Standardization 


10 


.1813** 




Item Subgroup Scores and 
Interquartile Range 


U 


.1891** 


.0078 


Item Subgroup Scores, 

interquartile Kange, ana rrouuci: 


21 


.2357** 


.0466* 


Item Subgroup Scores — Single 
Standardization 


10 


.1813** 




Item Subgroup Scores and 

Interquartile Deviations of 
3 . 5 or More 


11 


. 182A** 


.0011 


Itra Subgorup Scores, 

Interquartile Deviations of 
3.3 or More, and Product 


21 


.2063** 


.0239 


Icera Subgroup Scores — Single 
Standardized 


10 


.1813** 




Item Subgroup ard Difference 
Between Median Interquartile 
Deviation for Subject and 
Sample 


11 


.1815** 


.0002 


Item Subgroup, Difference Between 
Median Interquartile Deviation 
for Subject and Sample, and 
Product 


21 


.2096** 


.0281 
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Table 5 (Continued) 



Measures jj g E Increase 



Item Subgroup Scores — Single 
Standardized 

Item Subgroup Scores and 
Items Changed 



Item Subgroup Scores, Items 
Changed, and Product 



10 



1813** 



11 



.1826** 



0013 



21 



1937** 



0111 



Item Subgroup Scores — Single 
Standardized 



10 



1813** 



Item Subgroup Scores and 
Maximum Changes Per item 

Item Subgroup Scores, Maximum 
Changes Per Item, and Product 



11 



21 



1883** 



2012 



0070 



0129 



Item Subgroup Scores — Single 
Standardized 



10 



1813** 



Item Subgroup Scores and 
Effort on Test 



11 



1843** 



0030 



Item Subgroup Scores, Effort on 
Test, and Product 



21 



2078** 



02 3 '3 



Item Subgroup Scores — Single 
Standardized 



10 



1813** 



Item Subgroup Scores and 
Tiredness During Test 

Item Subgroup Scores, Tiredness 
During Test, and Product 



11 



21 



. 1877** 



2128** 



0064 



0251 



Item Subgroup Scorcs-~Single 
Standardized 



10 



1813** 



Item Subgroup Scores t.nd 
Computer Use 



11 



1813** 



0000 



Item Subgroup Scores, Computer 
Use, and Product 



21 



2218** 



0405 
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Measures q EE Increase 



Item Subgroup Scores — Single 
Standardized 

Item Subgroup Scores and 
Paragraph Comprehension 

Item Subgroup Scores, 

Paragraph Comprehension, 
and Product 



<.05; **£ <.01 



10 



.1813** 



11 



.1844** 



.0031 



21 



.1981** 



.0137 
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Figure Captions 

Flg:ure 1 , Internal-consistency reliability of double-standardized item 
subgroup scores, 

FigLure 2 . Internal-consistency reliability of single-standardized item 
subgroup scores* 

Fi gure 3 . Predictive validity of double-standardized item subgroup 
scores , 

Fi gure 4 . Predictive validity of single-standardized item subgroup 
scores . 
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